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Final  Technical  Report 

1991  SIAM  WORKSHOP  ON 
AUTOMATIC  DIFFERENTIATION  OF  ALGORITHMS: 
THEORY,  IMPLEMENTATION,  AND  APPLICATION 

Hilton  Hotel,  Breckenridge,  Colorado 
January,  6-8. 


The  workshop  was  attended  by  sixty-five  researchers,  of  whom  thirty  gave  half  hour 
lectures,  and  nine-teen  presented  posters.  This  was  the  first  scientific  meeting  devoted  to 
the  topic  of  automatic  differentiation  of  algorithms.  The  organizers,  Andreas  Griewank  and 
George  Corliss,  were  quite  successful  in  their  attempt  to  draw  participants  from  a  large 
variety  of  fields  and  institutions.  About  a  third  of  the  participants  and  speakers  came  from 
overseas.  The  response  of  the  participants  was  extremely  favorable,  with  many  expressing 
pleasure  at  the  breadth  and  quality  of  the  work  discussed. 

As  explained  by  Prof.  Masao  Iri  and  Prof.  George  Cybenko,  the  top  down  or  reverse 
mode  of  automatic  differentiation  was  already  utilized  in  the  early  seventies  -  by  Seppo 
Linnainmaa(1970)  for  the  estimation  of  rounding  errors  ,  and  by  Paul  Werbos  for  training 
neural  networks  by  backpropagation.  At  the  same  time  the  forward  mode  had  already 
been  implemented,  e.g.,  in  the  computer  language  PROSE,  co-authored  by  Joseph  Thames, 
who  presented  his  integrated  applications  package  at  this  workshop.  Several  other  speakers 
presented  automatic  differentiation  implementations,  including  precompilers  for  Fortran 
programs,  integrated  symbolic/numerical  environments  for  PC’s  ,  and  implementations  by 
overloading  in  C++,  ADA,  and  other  advanced  languages.  As  an  indication  of  the  very 
strong  interest  in  practical  implementations  and  applications,  we  note  that  throughout  the 
workshop  it  was  difficult  to  gain  access  to  one  of  the  two  AT  compatibles  with  modems:  the 
participants  were  demonstrating  their  PC  software  or  conducting  numerical  experiments  on 
their  home  computers! 

The  presentations  of  Prof.  Bruce  Char  and  Dr.  Victor  Goldman  were  particularly  valu¬ 
able  in  discussing  the  relationship  between  fully  symbolic  computer  algebra  and  the  more 
numerical  technique,  which  formed  the  primary  focus  of  this  workshop.  Both  acknowledged 
that  automatic  differentiation  is  significantly  more  efficient  in  certain  situations  and  advo¬ 
cated  a  merging  of  the  two  approaches  in  order  to  obtain  the  best  of  both  worlds  in  terms 
of  flexibility,  convenience,  and  efficiency. 

There  were  many  excellent  talks  on  large  scale  applications  -  in  particular  weather 
modeling,  oceanography,  petroleum  reservoir  modeling,  beam  tracing  in  optics,  satellite 
orbit  analysis,  and  mechanical  systems  simulation.  Some  of  the  speakers  delineated  very 
clearly  the  remaining  deficiencies  of  currently  available  automatic  differentiation  techniques 
in  comparison  to  handcoded  derivative  evaluation  programs.  From  the  lively  discussion  that 
followed,  it  can  be  expected  that  the  software  developers  present  will  accept  the  challenge 


of  closing  this  gap  in  efficiency  without  sacrificing  user  convenience. 

Several  speakers  pointed  beyond  the  traditional  core  concern  of  evaluating  point  deriva¬ 
tives  of  first  and  second  order  from  user-defined  programs.  They  proposed  the  exploitation 
of  automatic  differentiation  techniques  for  the  solution  of  various  computational  task,  e.g., 
concurrent  scheduling  on  advanced  architectures,  the  numerical  solution  of  differential  equa¬ 
tions  with  guaranteed  bounds,  the  efficient  calculation  of  Newton  steps,  and  the  parallel 
solution  of  unconstrained  optimization  problems.  It  became  clear  that  there  remain  many 
theoretical  and  practical  challenges,  for  example,  regarding  the  differentiation  of  multivari¬ 
ate  implicit  functions  arising  in  differential  algebraic  equations. 

In  many  formal  and  informal  discussions,  the  participants  raised  the  question  of  how 
potential  users  and  the  scientific  community  at  large  cam  be  made  aware  of  the  extremely 
promising  computational  techniques  presented  at  the  workshop.  Some  felt  that  the  name 
Automatic  Differentiation  sounds  too  mechanical  and  fails  to  indicate  the  wealth  of  intrinsic 
problems  and  ramifications.  Alas,  none  of  the  many  alternative  suggestion  seemed  to  gain 
instantly  widespread  acceptance.  The  participants  did  agree  on  the  importance  of  main¬ 
taining  the  momentum  of  the  woorkshop-for  example,  by  publishing  articles  in  professional 
news  letters  and  popular  scientific  journals.  The  organizers  view  the  publication  of  well- 
written  and  coherent  workshop  proceedings  as  a  very  important  first  step  in  popularizing 
automatic  differentiation  as  a  standard  computational  tool. 

The  size  and  format  of  the  workshop  appeared  fairly  optimal.  All  talks  were  presented 
in  the  same  lecture  room,  with  the  posters  lining  the  side  wall,  and  one  personal  computer 
each  placed  i.i  the  front  and  back.  Despite  the  unconventional  and  rather  tight  schedule, 
especially  on  Sunday,  the  attendance  was  generally  very  good,  and  no  more  than  a  handful 
of  attendants  left  before  the  final  lunch  on  Tuesday.  While  only  a  minority  of  the  attendants 
used  the  opportunity  to  ski  in  the  afternoons,  most  people  seemed  to  like  the  setting.  The 
timeslot,  early  January,  was  generally  considered  a  good  choice,  because  classes  had  not  yet 
started  or  only  just  begun,  so  that  academics  without  administrative  duties  could  get  away 
without  too  much  difficulties.  However,  on  account  of  the  low  temperatures  in  the  Rockies, 
early  January  is  not  the  prime  skiing  season,  and  one  should  be  able  to  secure  a  better 
deal  from  a  hotel.  The  Hilton  in  Breckenridge  had  apparently  so  far  little  experience  with 
groups  like  ours,  but  the  friendly  staff  made  every  effort  to  smoothen  out  minor  problems.  In 
particular,  they  were  very  accommodating  with  regards  to  late  cancellations,  room  changes 
etc. 

Even  though  many  of  this  year’s  posters  would  have  warranted  an  oral  presentation,  the 
organizers  believe  that  the  automatic  diffentiation  community  is  not  yet  strong  enough  to 
support  a  workshop  of  comparable  quality  and  originality  every  year.  Therefore  we  currently 
consider  January  1993  as  the  earliest  appropriate  date  for  the  second  workshop  on  the  same 
or  a  closely  related  topic.  During  the  intervening  time,  we  will  strive  to  propagate  automatic 
differentiation  through  minisymposia  and  short  courses  at  major  conferences.  Prof.  Weldon 
Lodwick,  University  of  Colorado  at  Denver,  took  good  care  of  the  computing  equipment 
this  time  and  has  offered  to  assist  with  all  local  arrangements  should  the  workshop  be  held 
again  in  Colorado. 


Andreas  Griewank,  George  Corliss 


Sunday,  January  8th,  p.m. 


1.00  Opening  remarks  by  Andreas  Griewank,  Argonne  National  Laboratory 


1.15  Prof.  Masao  Iri,  Tokyo  University 

History  of  automatic  differentiation  and  error  estimation 

1.45  Prof.  Louis  Rail,  University  of  Wisconsin 
Point  and  Interval  Differentiation  Arithmetics 

2.15  Dr.  Joseph  Thames,  Digital  Calculus  Corpoioticn,  CA 
A  new  paradigm  in  scientific  computing 

2.45  Prof.  George  Cybenko.  SirDa  Saarinen,  and  Randy  Bramlev 
CSRD,  University  of  Illinois  at  Urbana 

Neural  networks,  automatic  differentiation,  and  training  algorithms 

3.15  -  3.45  Coffee  Break 

3.45  Prof.  John  Burns,  Virginia  Polytechnic 
Relation  between  adjoinis  and  their  discretizations 

4.15  Dr.  Charles  Lawson,  JPL  CalTech 
Automatic  differentiation  of  inverse  functions 

4.45  Introduction  of  poster  presenters  and  titles 

5.00  Prof.  Harley  Flanders,  University  of  Michigan 

Symbolic  and  Automatic  Differentiation  of  Composite  Functions 

5.30  Prof.  Martin  Berz,  Michigan  State  University 

Infinitesimals,  higher  order  optical  aberrations,  and  the  supercollider 

6.15-7.15  Communal  Dinner 

7.30  Prof.  Talagrand,  Laboratoire  de  Meteorologie  Dynamique,  Paris 

The  use  of  adjoint  equations  in  numerical  modeling  of  the  atmospheric  circulation 

8.00  Dr.  Piyush  Shah,  Schlumberger  Well  Services,  Houston 

Application  of  Adjoint  Equations  to  Parameter  Estimation  in  Petroleum  Reservoirs 
and  Building  Structures 

8.30  Dr.  W.  Thacker,  Atlantic  Ocean  and  Meteorological  Laboratory,  Miami 
Solving  inverse  problems  in  oceanography  using  adjoint  equations 

9.00  Prof.  Navon,  Florida  State  University 

Derivation  and  verification  of  adjoints  for  a  3D  spectral  model  in  meteorology 


3 


Monday,  January  7th,  a.m. 


8.00  Dr.  Andreas  Griewank,  Argonne  National  Laboratory 
Automatic  Calculation  of  Jacobians  and  Newton  steps 

8.30  Dr.  Herbert  Fischer  and  Prof.  Klaus  Ritter,  Technische  Universitat  Miinchen 
Automatic  evaluation  of  first  and  second  derivatives  for  optimization 

9.00  Prof.  Laurence  Dixon,  Hatfield  Polytechnic 

The  impact  of  automatic  differentiation  on  (parallel)  optimization 

9.30- 10.00  Coffee  Break 

10.00  Dr.  David  Gay,  AT&T  Bell  Laboratories 
Differentiation  in  modeling  languages 

10.30  Prof.  Bruce  Char,  Drexel  University 

Computer  algebra  for  program  manipulation 

11.00  Prof.  Victor  V.  Goldman,  J.  Molenkamp,  and  J  .A.  van  Hulzen,  University  of  Twente 
Automatic  Generation  of  Efficient  Code  fo *•  Jacobians  and  Hessians  within  c  Com¬ 
puter  Algebra  Environment 

11.30- 5.00  Break 

5.00  Mr.  Don  Layne,  Martin  Marietta  Aeronautics 

Application  of  automatic  differentiation  and  self-validating  methods  in  satellite  simu¬ 
lations 

5.30  Dr.  Rudolf  Lohner,  Karlsruhe  University 

Use  of  symbolic  and  automatic  differentiation  in  self-validating  methods 

6.00  Prof.  George  Corliss,  Marquette  University 

Overloading  of  point  and  interval  Taylor  operators 

0.45-7.45  Communal  Dinner 

8.00  Prof.  Leigh  Tesfatsion,  Iowa  State  University 

Automatic  evaluation  of  higher  order  partial  derivatives  for  nonlocal  sensitivity  anal¬ 
ysis 

8.30  Dr.  Leo  Michelotti,  Fermi  National  Laboratory 

A  C++  Implementation  of  Differential  Algebra  for  Exploratory  Orbit  Analysis 

9.00  Dr.  Oscar  Garcia,  Forest  Research  Institute,  NZ 

Automatic  differentiation  in  maximum-likelihood  estimation  of  growth  model  parame¬ 
ters 


U 


Tuesday,  January  8th,  a.m. 


8.00  Prof.  Youri  Evtushenko,  USSR  Academy  of  Sciences 

Fast  automatic  differentiation  in  optimal  control  and  partial  differential  equations 

8  30  Dr.  Vladimir  Mazourik,  USSR  Academy  of  Sciences 

Integration  of  automatic  differentiation  into  a  numerical  library  for  PC’s 

9.00  Dr.  Koichi  Kubota,  Keio  University,  Japan 

PADRE2,  a  FORTRAN  precompiler  yielding  error  estimates  and  second  derivatives 

9.30  Mr.  Jim  Horwedel,  Oak  Ridge  National  Laboratory 

Gress,  a  preprocessor  for  sensitivity  studies  on  FORTRAN  codes 


10.00-10.15  Coffee  Break 


10.15  Dr.  Christian  Bischof,  Argonne  National  Laboratory 
On  the  road  to  parallelism  in  automatic  differentiation 

10.45  Dr.  Edgar  Soili,  Centre  d’Etudes  Nuclanes  de  Saclay 
User’s  experience  with  FORTRAN  precompilers 

11.15-12.00  Discussion  of  software  and  future  developments. 


Titles  of  Posters 


1.  Y.F.  Chang 

A  variable-order  (10-600)  Taylor  series  method  for  solving  ODE’s,  with  integration 
steps  almost  equal  to  the  radii  of  convergence 

2.  Bruce  Christianson,  Hatfield  Polytechnic 
Automatic  Hessians  by  reverse  accumulation  in  Ada 

3.  Robert  Corless,  University  of  Western  Ontario 

Automatic  differentiation  in  validated  defect  control  for  ordinary  differential  equations 

4.  Hans  Christoph  Fischer,  University  of  Karlsruhe 
Differentiation  arithmetic  and  applications  in  Pascal-XSC 

5.  William  W.  Hager,  University  of  Florida 
Dual  active  sets  and  constrained  optimization 

6.  David  Hill,  Temple  University  and  Larry  Rich,  The  Mathematical  Software  Group 
MATLAB  +  DERIVATIVES  =  OPPORTUNITIES 
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7.  James  Hu,  University  of  Kansas 

Optimizing  computational  graphs  for  automatic  differentiation. 

8.  Baker  Kearfott,  University  of  Southwestern  Louisiana 

Use  of  intermediate  quantities  in  expression  evaluation  to  improve  the  behavior  of 
interval  iteration  for  nonlinear  systems 

9.  Max  E.  Jerrell,  Northern  Arizona  University 
Automatic  Differentiation  and  Estimation 

10.  David  Juedes,  Iowa  State  University 

ADOL-C,  A  package  for  the  automatic  differentiation  of  algorithms  written  in  C/C+  + 

11.  Thomas  Nehrkorn,  Atmospheric  and  Environmental  Research,  Inc. 

Optimization  of  model  parameters  using  the  adjoint  method 

12.  Richard  Neidinger,  Davidson  College 

Automating  arbitrary  order  derivatives  for  multivariable  functions 

13.  Dinesh  K.  Pai,  Cornell 

Automatic  differentiation  in  simulation  and  control  of  mechanical  systems 

14.  Flcrian  Potra,  University  of  Iowa 

Differentiation  of  kinematic  constraints  in  multibody  dynamics 

15.  Marcela  Rosenblun,  Rice  University 

Using  automatic  differentiation  in  constrained  optimization 

16.  Nicole  Rostaing,  University  of  Nice-Sophia  Antipolis 
Automatic  generation  of  partial  derivatives  using  REDUCE 

17.  Herbert  Warsitz,  Technische  Universitat  Mvinchen 

Solving  constrained  and  unconstrained  non-linear  optimization  using  PADMOS 

18.  Toshinobu  Yoshida,  University  of  Tokyo 

1.  ADDS  (Automatic  Derivative  Derivation  System) 

2.  Why  CAN  we  use  second  order  learning  methods  for  large-scale  neural  networks? 

19.  Xaiolei  Zou,  Florida  State  University 

Variational  f-dimensional  data  simulation  using  an  adiabatic  version  of  the  NMC 
spectral  model 
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